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COOPERATIVE SYSTEM FOR MEASURING ELECTRONIC MEDIA 

Cross-Reference to Related Applications 

5 This application is a continuation of pending United States Patent Application 

Serial Number 08/880,371, filed June 23, 1997, incorporated by reference herein. 

Field of the Invention 

The present invention relates generally to a system for measuring a population's 
10 exposure to and interactions with electronic media (hereinafter, "electronic media measurement 
systems"), and more particularly, to a cooperative electronic media measurement system using 
§»* media handlers to extract information from, or otherwise obtain information about, presented 
^ media objects, including identification tags, if present, for collection by software agents on behalf 
tti of a centralized media research facility. 



Background of the Invention 

The success of any advertising campaign depends on the accurate placement of 
advertisements within media, and the verification that specific advertising messages were 
presented in accord with a predefined media plan. Generally, an advertising campaign is targeted 
for one or more segments of a population, with media planners determining the best media 
vehicles to reach the target audience. In this manner, the advertiser seeks to find the most 
efficient media to minimize the cost to deliver a desired audience. 

Thus, prior to executing a given advertising campaign, media planners use 
syndicated research, such as Nielsen ratings, to determine the best media vehicles to reach a 
2 5 target audience. In addition, media planners utilize other information sources to research and 
compare the costs associated with reaching an audience through each available media vehicle. 
During a given advertising campaign, it is helpful to measure the target audience's exposure to 
the advertising messages, since media planners might make corrections in order to optimize the 
execution of the media plan. Likewise, after a given advertising campaign, media planners often 
30 analyze the execution of the campaign to confirm that the advertising messages reached the 
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targeted audience to determine the accuracy of the campaign's messages in reaching the targeted 
audience. 

As audiences have fragmented, due to the increasing number of available channels 
and online options, it has become increasingly challenging for media planners to determine 
5 which media vehicles provide the best avenue to a given audience. The Internet, in particular, 
provides advertisers with many media options and is becoming ubiquitously available in an 
expanding variety of personal electronic devices, far beyond its initial limited availability to users 
via computer terminals and desktop computers. As with other media, advertising has become an 
important part of Internet revenue models. Much of the Internet's value to the advertising 
10 community is due to its enormous and evolving diversity of advertising formats, including the 
banner ad and Java applets, and its capability to deliver customized and relevant advertising to 
j - end users. For a more detailed discussion of advertising media, see D. Jugenheimer et aL, 
Advertising Media Strategy and Tactics (W.C.B. Brown & Benchmark, 1992), incorporated by 
reference herein. 

Thus, the Internet provides an efficient mechanism for matching the advertising 
message to the appropriate segment of the audience. Such diverse advertising formats, however, 
present challenges for measuring a population's exposure to and interactions with such 
advertisements. While the success of the Internet can be attributed, in large part, to its open 
media standards that permit the creation and delivery of content having diverse formats across 
St) many platforms, there is currently no user-centric system capable of adequately measuring the 
diverse media formats across the growing variety of Internet-enabled consumer platforms, 
consistent with the needs of the advertising community. 

Generally, a given population's exposure to and interactions with media is 
measured by knowing the television channels and other information sources that the members of 
25 the population select. This can be performed either as a census, where the choices of the entire 
population are collected, or as a sample, where a statistically valid sub-population or panel is 
chosen to represent the entire population. Nielsen Media Research, for example, uses a panel of 
households, known as "Nielsen Families," for measuring television viewing. Such panels enable 
research companies to correlate demographics, such as age, gender, income and education, with 
3 0 choice of content. 



m 

rsil 



2 



PATENT 



Docket No. 1600-2 US2 



Conventional content frequently contains, or is associated with, metadata that 
provides information about the content. For example, many broadcasters transmit information 
with conventional programming to help identify the content, for example, by program and 
episode. Nielsen, for example, extracts such accompanying information for measurement 
5 purposes to track the programs viewed by certain members of a panel. In addition, smart 
electronic program guides use such accompanying information to help individuals or their agents 
find content of interest. 

Similarly, the World Wide Web Consortium (W3C), has endorsed the Platform 
for Internet Content Selection (PICS), which is an open standard for tagging information and 
10 coding content on the Internet. The PICS standard is designed to allow software to automatically 
filter content that individuals choose not to receive, such as violent content, according to a 
u ratings system. The PICS standard provides parents and other individuals with the ability to 

O select categories of content that can be automatically blocked, in a similar manner to V-chip 

O 

fy technology, for conventional programming. While the PICS standard allows an entire web site or 
fejp static pages to be rated, the PICS standard does not permit tagging content on an object level. 
2* In addition, traditional electronic advertising, such as television and radio 

m advertisements, have unique identification codes, or Industry Standard Commercial Identification 
(ISCI) codes, which are used for handling, broadcasting, storing and retrieving commercials. 
Under the ISCI standard, an ISCI alpha prefix and an ISCI numeric code identify each 
commercial. An ISCI prefix is assigned by ISCI to national and regional advertisers and 
advertising agencies. The ISCI code may be used in any manner, at the discretion of the prefix 
owner, provided the code consists of four letters followed by four numbers. Although ISCI codes 
are not presently encoded as computer readable data with each advertisement, they might evolve 
to do so for Internet advertising, to better manage advertising on the Internet. 

2 5 There exists both "pull" and "push" models for delivering Internet content. On 

traditional web sites, individuals "pull" content by browsing. These web sites can use tools to 
analyze the "hits" to their sites in real-time. Additionally, there exist "push" models of content 
delivery, such as provided by PointCast™. PointCast™ is a webcasting service that "pushes" or 
streams a variety of information, including editorial and advertising content, to a receiving 

3 0 software component, such as their proprietary screensaver, or Microsoft's Internet Explorer 
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browser, version 4.0. Presently, each PointCast™ subscriber self-reports demographic 
information. Therefore, PointCast™ can provide advertisers with user-centric information about 
advertising exposure. Self-monitoring, however, is contrary to advertising industry guidelines, 
which express a preference for measurement by a disinterested third party. In addition, the 
5 PointCast™ system is limited to measuring only electronic media distributed by PointCast™. 

Even assuming that an independent auditor verified such site-centric 
measurements, the measurements often do not accurately reflect the activity of individuals. For 
example, many of the "hits" on a web site are associated with electronic agents that perform 
functions on the Internet on behalf of individuals. Examples of such electronic agents include 
10 web robots, issued by search engines such as those provided by Infoseek Corp. to index the 
contents of the Internet, and personal agents that automatically retrieve information from the 

j** Internet that matches the specified preferences of an individual. Thus, such electronic agents 

O 

Q increase the traffic count of the respective web site, as they are not necessarily representative of 

nj 

•4 an individual viewing Internet content. For example, an agent might download the entire 
Ife contents of a site, while the user only views a single article. Proxy servers, on the other hand, 
iy which cache or copy Internet content to a local server or hard disk drive for subsequent access, 
can decrease the traffic count of a given web site. Proxy servers are used to reduce access time 

2 by storing a copy of information that was recently downloaded from a site. Thus, upon a 
I-* subsequent request, the information can be delivered from the local server rather than the Internet 
JSp without the knowledge of the web site traffic counter. 

While conventional electronic media measurement systems, such as Nielsen 
Media Research's PeopleMeter™, have successfully measured traditional media, such as 
television and radio, such systems are not easily extendable to the Internet environment. In 
addition, the site-centric measurement approaches discussed above have proven unsatisfactory. 
25 In order to accurately measure a population's exposure to and interactions with such electronic 
media, a user-centric measurement approach is needed which is based on a panel chosen to be 
statistically representative of the total population of interest. Current user-centric Internet 
measurement systems, however, such as the NPD Group's PC Meter™, are based on interception 
and interpretation of electronic media presented to members of a panel. Such interception 

3 0 techniques, however, rely on observing calls by software applications to the operating system and 
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require privileged access into operating system internals. Furthermore, PC Meter™ is currently 
limited to household users of the Windows™ operating system, which may not be statistically 
representative of the total population of interest. For a more detailed discussion of the PC 
Meter™ system, see PCT Published Application Number WO 96/41495. 
5 It is believed that observing operating system internals will become increasingly 

challenging, if not impossible, with the trend towards more secure operating systems and 
communication security. Windows NT™ from Microsoft™, for example, implements a 
concentric ring structure of ascending privilege with an outermost ring of lowest privilege and an 
innermost ring of highest privilege, from which applications are excluded, based on the processor 
10 ring architecture specified by Intel Corporation. As security services become more available to 
Internet applications, both for computer-to-computer communications and application-to- 
application communications, much of this internal traffic will be encrypted. In addition, such 
O operating system monitoring techniques will be challenging to implement within the many 
JU proprietary implementations of Internet-enabled devices, such as WebTV™. Even assuming that 

ife such user-centric measurement systems are successful in obtaining access to these 

M 

flj communications, it is very challenging to understand what the intercepted messages mean. 

^. A recent industry article indicates that a new company, Relevant Knowledge Inc., 

J}jj of Atlanta, Georgia, has developed a real-time approach to compete with the PC Meter™ system. 
Although Relevant Knowledge Inc. did not comment publicly for the article, it does not appear 

SO that Relevant Knowledge Inc. is using a cooperative approach. Rather, it appears that Relevant 
Knowledge Inc. is monitoring information, using interception and interpretation, and leveraging 
the communication capabilities of the Internet to distribute their monitoring software to their 
panel members and to collect data in real-time. 

As apparent from the above-described deficiencies with conventional electronic 

25 media measurement systems, a need exists for a universal system for measuring electronic media 
having diverse formats, including television, radio, Internet, and online services, across a 
plurality of platforms. A further need exists for a cooperating system that extends the open 
media standards of the Internet to measure a population's exposure to and interactions with such 
electronic media. Yet another need exists for a system to measure traditional television, radio, 

3 0 cable television, digital satellite programming and advertising delivered to households that use 
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Internet-enabled computers and appliances for viewing, listening and interacting with such 
content. 

Summary of the Invention 

Generally, according to one aspect of the invention, media information presented 
to a panel member by means of a panel member-computing device is measured by a media 
research controller for subsequent reporting to one or more research customers. The panel 
member-computing device may receive the media information by means of a network 
connection, or from one or more local sources, such as prerecorded media obtained from a CD- 
ROM or DVD player, or may generate the media objects in real-time, or a combination thereof. 
The media research controller registers advertisements and other media for subsequent 
measurement and provides a unique identification tag that may be added to, or associated with, 
the existing media object for identification purposes. In addition, the present invention extracts 
information from, or otherwise obtains information about, presented media objects, including 
metadata or other information associated with a given media object, for later collection by the 
media research controller even when the media object has not been previously registered and 
tagged by the media research controller. Thus, a panel member's exposure to and interactions 
with all electronic media is measured, regardless of whether the media has been previously 
registered or tagged for identification purposes. The panel members are preferably chosen for 
their demographics and have agreed to participate in a research panel to have their electronic 
media measured. 

The media research controller preferably assigns one or more software agents, 
hereinafter referred to as research data collection (RDC) agents, to measure each panel member's 
exposure to and interactions with electronic media. Thus, each research data collection agent 
serves as an intermediary between the local environment of a panel member and the central 
media research controller. In one embodiment, a research data collection agent is associated with 
each computing device utilized by a given panel member. Thus, a single panel member might be 
assigned multiple research data collection agents, if required, to measure the panel member's use 
of electronic media across multiple devices. Furthermore, a single research data collection agent 
may serve a plurality of panel members utilizing the same computing device. 
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According to a further aspect of the invention, cooperative media handlers are 
utilized by the panel member-computing device to present media to a panel member and to 
extract information from, or otherwise obtain information about, the media objects, including 
identification tags, if present, for collection by the research data collection agents. As used 
herein, the term cooperating system means a system that relies on the media handler software 
which presents the media to cooperate by gathering and developing information about media 
activities of end users, and transmitting such activity information to the media research 
controller, either directly or indirectly via the research data collection agents, in effect acting as 
agents of the media research controller, as opposed to conventional approaches of intercepting 
and interpreting the media activities of end users. 

As used herein, the term media handler includes persistent software components 
which extend the capabilities of a software application or operating system to present media 
objects of a particular media type to an individual and autonomous software components, such as 
Java applets, which may only temporarily extend the capabilities of the host to present media. In 
addition, the term media handler includes software applications that generate media experiences 
in real-time, such as video games, and resident software components, such as the PointCast™ 
agent, which present media to end users. In addition, the term media handler includes those 
portions of an Internet browser, also called "viewers" and "plug-ins", that are capable and 
responsible for decoding specific media types, such as JPEG images, and using the resources of 
their host to present the media to the end user. The media handlers may be mobile, moving from 
one host computer to another, or stable, anchored to one host. 

Thus, in addition to their primary function of presenting media, the cooperative 
media handlers also serve as software agents for the research data collection agents, by gathering 
and deriving relevant information about the media presentation on behalf of the research data 
collection agents and then sending this information to the research data collection agents. The 
media handlers are in the best position to provide information about the media objects presented 
to the panel members, since the media handler can decode the particular media type, and 
determine what is presented to the individual, when it is presented and how the individual 
interacts with the object. In this manner, the media handler can report on a panel member's 
exposure to and interactions with a media object, such as zooming in on a particular feature of a 
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media object or rotating the object, and the source of the control signals, such as a specific 
individual or process. In the case of stable, installed media handlers, such as video games or 
screensaver clients; or in the case of mobile, transient media handlers, such as a Java applet 
advertisement, they become cooperative through the implementation of an application 
programming interface (API), and communicate to the research data collection agents via their 
host. In all of these instances, the media handler's host might contribute contextual information, 
such as the web site of origin, to the media handler along with the media object. 

The hosts of the research data collection agent and the cooperative media handlers 
preferably provide them with necessary computational resources, such as processor cycles, 
memory and communication. In the illustrative embodiment, an Internet browser software 
product, such as Netscape Navigator™ or Microsoft Internet Explorer™, resident on the panel 
member-computing device, serves as the host for both the research data collection agent and the 
cooperative media handler. In alternate embodiments, the host for one or both of the research 
data collection agent and the cooperative media handlers may be embodied as an operating 
system or a virtual machine, such as the Java virtual machine. The research data collection 
agents and the cooperative media handlers (the hosted processes) preferably use an application 
programming interface (API) to define the function calls which the hosted processes and their 
hosts use to communicate and share resources and services. In this manner, different companies 
can develop interoperable research data collection agents, cooperative media handlers and hosts. 

When electronic media is received by a cooperative media handler, the media 
handler automatically extracts information from, or otherwise obtains information about, the 
media, including an identification tag, if present, for transmission to a research data collection 
agent. In addition to the extracted identification tag, if present, the cooperative media handler 
preferably transmits any content metadata included in or associated with the media object, an 
indication of the media handler's identification number and any contextual information which 
has been made available to the media handler by its host, including program, episode, version, 
and source information for television and radio programming, or a source computer's domain 
name or IP address for a web site, and content rating information, such as PICS, if available. 

A research data collection agent preferably commences tracking of a panel 
member, and begins collecting such transmissions from the cooperative media handlers, when a 
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panel member enters the scope of the research data collection agent and has been identified and 
authenticated, for example, following a log on procedure. Similarly, the research data collection 
agent suspends tracking of a panel member when the person leaves the scope of the remote media 
research agent, such as following a log off procedure. The research data collection agent (i) 
creates log entry objects from transmissions received from the cooperative media handlers and 
places the log entry objects into an unfiltered media queue, (ii) confirms the integrity of the 
messages and filters out unnecessary log entry objects from the unfiltered media queue to create a 
filtered media queue, (iii) creates dispatch objects using objects from the filtered media queue 
and places created dispatch objects into a dispatch queue, and (iv) transmits dispatch objects 
from the dispatch queue to the media research controller, when resources are available. 

Identification tags are preferably placed at periodic intervals throughout the 
duration of continuous media, such as audio and video, or on an associated data channel. Since 
identification tags might be utilized by hostile software to automatically remove advertising 
media objects, the identification tags are preferably placed in both the advertising, as well as the 
associated advertising-supported media objects to discourage such automatic removal. In one 
preferred embodiment, "real" identification tags are placed in media that is to be measured (and a 
fraction of media that is not to be measured), and "dummy" identification tags are placed in other 
content. The research data collection agents can preferably distinguish real identification tags 
from dummy identification tags, and, if desired for efficiency purposes, can only return 
measurements about registered media objects containing real identification tags to the media 
research controller. 

A more complete understanding of the present invention, as well as further 
features and advantages of the present invention, will be obtained by reference to the following 
detailed description and drawings. 
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Brief Description of the Drawings 

FIG. 1 is a schematic block diagram illustrating a suitable communications 
network for interconnecting a panel member with one or more content providers and a media 
research controller of an electronic media measurement system; 

FIG. 2 illustrates the communications and cooperative relationship between the 
various entities shown in FIGS. 1, 3 and 4; 

FIG. 3 is a schematic block diagram of the media research controller of FIG. 1; 

FIG. 4 is a schematic block diagram of a panel member-computing device of FIG 

l; 

FIG. 5 illustrates a sample table from the panel member database table of FIG. 2; 
FIG. 6 illustrates a sample table from the registered media database table of FIG. 

2; 

FIG. 7 illustrates a sample table from the encryption keys database table of FIG. 2; 

FIG. 8 illustrates a sample table from the log entry database table of FIG. 2; 

FIG. 9 illustrates a sample table from the local panel member database table of 

FIG. 3; 

FIG. 10 illustrates a sample table from the local encryption keys database table of 

FIG. 3; 

FIG. 11 illustrates a sample log entry object from the media queues of FIG. 3; 

FIG. 12a is a flow chart describing an exemplary tag registration and encoding 
process as implemented by the media research controller of FIG. 3; 

FIGS. 12b and 12c illustrate an exemplary identification tag format definition for 
a real identification tag and a dummy identification tag, respectively; 

FIG. 13 is a flow chart describing a data importing process as implemented by the 
media research controller of FIG. 3; 

FIG. 14 is a flow chart illustrating a reporting process as implemented by the 
media research controller of FIG. 3; 

FIG. 15a provides an overview of the processes shown in FIGS. 15b through 15f 

on the media queues of FIG 4; 
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FIG. 15b is a flow chart describing a logging process as implemented by the 
research data collection agent of FIG. 4; 

FIG. 15c is a flow chart describing a filter process as implemented by the research 

data collection agent of FIG. 4; 

FIG. 15d is a flow chart describing a tag decoding subroutine as implemented by 

the research data collection agent of FIG. 4; 

FIG. 15e is a flow chart describing a create-dispatch object process as 
implemented by the research data collection agent of FIG. 4; 

FIG. 15f is a flow chart describing a dispatch process as implemented by the 
research data collection agent of FIG. 4; 

FIG. 16 is a flow chart describing a suitable host tracking process as implemented 
by the panel member-computing device of FIG. 4; and 

FIG. 17 is a flow chart describing a cooperative media handler process as 
implemented by the panel member-computing device of FIG. 4. 

Detailed Description 

FIG. 1 shows an illustrative network environment for transferring media 
information, such as video, audio and data, from one or more content providers 110, 120, to a 
panel member 205 operating a panel member-computing device 400, discussed further below in 
conjunction with FIG. 4, over one or more external networks, such as network 130. In an 
alternative or supplemental embodiment, discussed further below, the panel member-computing 
device 400 may be configured to receive media information from one or more local sources, such 
as prerecorded media obtained from a CD-ROM or DVD player, or to generate media objects in 
real-time, for example, by means of a video game generating an advertising image with the name 
of the panel member for placement on a wall inside a virtual world, or a combination thereof. 
According to a feature of the present invention, the media information presented to the panel 
member by means of the panel member-computing device 400 is measured by a media research 
controller 300, discussed further below in conjunction with FIG. 3, for subsequent reporting to 
one or more research customers 150. It is noted that the reports may be generated by the media 
research controller 300 in real time, or historically, or both. 
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The external networks 130 shown in FIG. 1 include the Internet, the Public 
Switched Telephone Network ("PSTN") and networks for the delivery of radio and television 
programming, such as the Digital Satellite Service ("DSS™"), cable television ("CATV") and 
other over-the-air transmission networks for broadcast television, radio and satellite 
communications. The PSTN, as used herein, includes the combination of local and long distance 
wire or wireless facilities and switches, as well as cellular network systems. The Internet, as 
used herein, includes the World Wide Web (the "Web") and other systems for storing and 
retrieving information using the Internet. 

According to a feature of the present invention, the media research controller 300 
registers advertisements and other media for subsequent measurement and provides a unique 
identification tag that may be added to, or associated with, the existing media object for 
identification purposes. In addition, the present invention extracts information from, or 
otherwise obtains information about, presented media objects, including metadata or other 
information associated with a given media object, for later collection by the media research 
controller 300 even when the media object has not been previously registered and tagged by the 
media research controller 300. In this manner, the present invention measures a panel member's 
exposure to and interactions with all electronic media, regardless of whether the media has been 
previously registered or tagged for identification purposes. The panel members are preferably 
chosen for their demographics and have agreed to participate in a research panel to have their 
electronic media measured. The panel is preferably constructed so that it is representative of and 
projectable to the entire population. It is noted that if a census is desired, as opposed to a panel, 
the panel simply consists of the entire population of interest. 

ENTITY RELATIONSHIPS AND INTERFACES 

As shown conceptually in FIG. 2, the present invention preferably consists of a 
number of cooperating entities to measure electronic media presented to a panel member 205 by 
means of the panel member-computing device 400, namely, a centralized media research 
controller 300, a research data collection agent 1500 and a cooperative media handler 1700. As 
previously indicated, the media research controller 300 measures a panel's exposure to and 
interactions with electronic media for subsequent reporting to one or more research customers 
150. As used herein, the term cooperating system means a system that relies on the media handler 
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software which presents the media to cooperate, by gathering and developing information about 
media activities of end users, and transmitting such activity information to the media research 
controller, either directly or indirectly via the research data collection agents, in effect acting as 
agents of the media research controller, as opposed to conventional approaches of intercepting 
5 and interpreting the media activities of end users. 

To this end, the media research controller 300 preferably assigns one or more 
software agents, referred to as research data collection agents 1500, discussed further below in 
conjunction with FIG. 15, to measure each panel member's exposure to and interactions with 
electronic media. Thus, each research data collection agent 1500 serves as an intermediary 
1 0 between the local environment of a panel member 205 and the central media research controller 
300. In one embodiment, a research data collection agent 1500 is associated with each 
u computing device 400 utilized by a given panel member 205. Thus, a single panel member 
might be assigned multiple research data collection agents, if required to measure the panel 
member's use of electronic media across multiple devices. Furthermore, a single research data 
collection agent 1500 may serve a plurality of panel members utilizing the same computing 
device 400. It is not necessary for the research data collection agent 1500 to be proximal to the 
associated panel member or the panel member's physical environment. Since the research data 
collection agent 1500 is user-centric, the agent 1500 might be a component of the person's net- 
centric environment and be activated as a side effect of the person obtaining access to network 

3D resources, or the agent 1500 may reside on a smartcard carried by the individual. 

Mi 

In addition, the cooperative media handlers 1700, discussed below in conjunction 
with FIG. 17, are utilized by the panel member-computing device 400 to present media to the 
panel member 205 and to extract information from, or otherwise obtain information about, 
presented media objects, including identification tags, if present, for collection by the research 

2 5 data collection agents 1500. Thus, in addition to their primary function of presenting media, the 

cooperative media handlers 1700 also serve as software agents for the research data collection 
agents 1500, by gathering and deriving relevant information about the media presentation on 
behalf of the research data collection agents and then sending this information to the research 
data collection agents. It is noted that while some of the media handlers might be implemented 

3 0 as cooperative media handlers 1700, as described herein, the present invention is not dependent 
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upon all of the media handlers being implemented in a cooperative manner. However, the extent 
of research data collection agent's 1500 ability to measure a panel member's exposure to and 
interactions with electronic media is limited to media presented by means of a cooperative media 
handler 1700. 

As used herein, a software agent is a software component that has the ability and 
authority to act on behalf of a controlling entity, generally instantiated as a process on a host 
computer. Each agent's controlling entity provides the agent with resources, such as encryption 
keys and a digital certificate for identification, and delegates specific tasks to the agent. The 
software agents manage these tasks, and operate in an autonomous or semi-autonomous mode 
with respect to its controlling entity. The software agents may be mobile, moving from one host 
computer to another, or stable, anchored at one host. 

The software agent's host preferably provides the software agent with necessary 
computational resources, such as processor cycles, memory and communication. In the 
illustrative embodiment, discussed further below, an Internet browser software product, such as 
Netscape Navigator™ or Microsoft Internet Explorer™, resident on the panel member-computing 
device 400, serves as the host 1600 for the research data collection agent 1500 and the host 1600' 
for the cooperative media handler 1700 and provides them with the required computational 
resources. In alternate embodiments, the host for one or both of the research data collection agent 
and the cooperative media handlers may be embodied as an operating system or a virtual 
machine, such as the Java virtual machine. The research data collection agent 1500 preferably 
communicates with the media research controller 300 by means of the host's secure 
communication capabilities, such as SSL or S/MIME. In addition, the research data collection 
agent 1500 relies on its host 1600 to pass through data sent by the cooperative media handler 
1700 for collection by a research data collection agent 1500. 

The communications between the various entities are preferably sufficiently 
encrypted to protect the privacy of the panel members 205, the proprietary information and 
competitive interests of the media research controller 300 and the validity of the collected data, 
as would be apparent to a person of ordinary skill Furthermore, to protect the privacy of non- 
panel members, the cooperative features of the hosts 1600, 1600', as described herein, are 
preferably initially disabled when distributed to end-users, and are only subsequently enabled 
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when specifically authorized by a panel member who has agreed to participate in a research 
panel. 

In some embodiments, the research data collection agent 1500 is not proximal to 
the panel member(s) it tracks, or the panel member's physical environment. For example, the 
5 research data collection agent might be hosted by the central computer of an online service. In 
other embodiments, the research data collection agent is hosted separately from the cooperative 
media handlers. For example, the research data collection agent 1500 might be located in a 
wristwatch worn by the panel member. Thus, the host services and resources may reside locally 
or be accessed by means of a network connection, or a combination thereof, as would be apparent 
10 to a person of ordinary skill. It is noted that the research data collection agents 1500 and the 
cooperative media handlers 1700 might typically share the same host or might be hosted 
separately, for example, in an embodiment where the media research data collection agent 1500 

■Pi 

S associated with a panel member 205 is resident on a smartcard or wristwatch carried by an 

■KSSt' 

individual. 

CP 

fil> As illustrated in FIG. 2, the present invention relies on a number of cooperative 

S3 

H! interfaces between the various entities. A media object is presented to a panel member 205 by 
: the cooperative media handler 1700 using a conventional bi-directional user interface 210. The 
fU cooperative media handler 1700 transmits identification tags and other information extracted 

PS s 

LI from, or otherwise obtained about, media objects for collection by a research data collection 
Sw agent 1500, by means of a cooperative interface 220. Upon receipt of an extracted identification 
tag or other obtained information from a media handler 1700, the research data collection agent 
1500 sends a message to the media research controller 300, preferably over an external network 
130, containing the extracted identification tag and other obtained information. 

In addition, as previously indicated, the research data collection agents 1500 and 

2 5 the cooperative media handlers 1700 also require interfaces to their respective hosts 1600, 1600'. 

In a preferred embodiment, the research data collection agents 1500 and the cooperative media 
handlers 1700 (the hosted processes) use an application programming interface (API) to define 
the function calls which the hosted processes and their hosts, such as the host 1600, use to 
communicate and share resources and services. In this manner, different companies can develop 

3 0 interoperable research data collection agents, cooperative media handlers and hosts. 
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ENTITY FUNCTIONS 
As previously indicated, when electronic media is received by a cooperative 
media handler 1700 for presentation, the media handler 1700 automatically extracts information 
from, or otherwise obtains information about, presented media object, including an identification 
5 tag, if present, and other relevant information, as available, for transmission to a research data 
collection agent 1500. In addition to the extracted identification tag, if present, the cooperative 
media handler 1700 preferably transmits any content metadata included in or associated with the 
media object, including program and episode information; an identification of media handler and 
its host; any contextual information which has been made available to the media handler 1700, 
10 such as an indication of the media source for television and radio programming, or a source 
computer's domain name or IP address for a web site; any presentation information or user 
interaction information, and content rating information, such as PICS, if available. Each of these 
3 information types is discussed further below in conjunction with FIG. 8. 

J** A research data collection agent 1500 preferably commences tracking of a panel 

fib member, and begins collecting such transmissions from the cooperative media handlers 1700, 

"4 

ffj when a panel member enters the scope of the research data collection agent 1500 and has been 

* identified and authenticated, for example, following a log on procedure. Similarly, the research 

HI data collection agent 1500 suspends tracking of a panel member 205 when the person leaves the 

U scope of the remote media research agent 1500, such as following a log off procedure. The 

fefc research data collection agent (i) creates log entry objects from transmissions received from the 

If* 

cooperative media handlers and places the log entry objects into an unfiltered media queue, (ii) 
confirms the integrity of the messages and filters out unnecessary log entry objects from the 
unfiltered media queue to create a filtered media queue, (iii) creates dispatch objects using 
objects from the filtered media queue and places created dispatch objects into a dispatch queue, 

25 and (iv) transmits dispatch objects from the dispatch queue to the media research controller, 
when resources are available. 

The cooperative media handlers 1700 preferably are not aware of whether or not a 
research data collection agent 1500 is present and measuring the content presented to a panel 
member, for the same reason that the identities of Nielsen families are carefully concealed from 

3 0 the media sources that Nielsen measures, to prevent manipulation of the content presented to the 
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panel members. Thus, the cooperative media handlers 1700 preferably transmit information 
extracted from, or otherwise obtained about, presented media objects, for collection by a research 
data collection agent 1500, whether or not a research data collection agent 1500 is present. For 
efficiency purposes, however, when the same host 1600 is hosting both the research data 
5 collection agent 1500 and media handler 1700, the host may inhibit the cooperative media 
handler 1700 from transmitting information extracted from or obtained about media objects when 
a research data collection agent 1500 is not present, is not in a tracking mode or the panel 
members are not in scope. 

According to a feature of the present invention, the media handlers 1700 thus 

10 cooperate with the research data collection agents 1500 by gathering and deriving relevant 
information about the media presentation on behalf of the research data collection agents and 

u then sending this information to the research data collection agents. The media handlers 1700 

2 are in the best position to provide information about the media objects presented to the panel 
RJ members 205, since the media handler is responsible for decoding or generating specific media 
|f objects, they determine what is presented, how it is presented, when it is presented and how the 
J-jj panel member 205 interacts with the presented media. Additionally, the media handlers 1700 

iH 

* can extract metadata from the media object and glean contextual information about the media 

jftl object from the media handler's host. Through their cooperation and assistance, the research 
data collection agent can gain access to information that might not be available in any other way. 

St) Since the media research controller 300 can measure a panel's exposure to and 

interactions with the cooperative media handlers 1700 themselves, in addition to exposure to and 
interactions with the underlying media objects of interest, providers of media handlers are 
motivated to develop media handlers which cooperate with the software processes of the present 
invention, and thereby ensure that the media handlers provide the required functionality. In this 

25 manner, media handler developers can receive custom research reports to analyze the panel's 
exposure to and interactions with such media handlers. 

MEDIA TERMINOLOGY 
A media type is a means to represent media information, such as an image or a 
sound. For example, the PNG (Portable Network Graphics) file format is a media type for 

3 0 representing computer images. As used herein, a media object is an item or instance of a media 
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type. Currently, there are hundreds of media types in use. Typically, media types are 
implemented as files, and increasingly as objects, and allow for some means of attaching 
metadata, or information about the data. In some cases, the metadata is implemented as a text 
field that resides in a header preceding the content data and can be extracted prior to decoding the 
5 content data. Generally, extraction of the header data is not computationally intensive, and can 
take place prior to or during the decoding and display or playback of the content. 

Every media type must have a corresponding media handler, located at the end- 
user's client side to present a given media object to an individual. As used herein, the term media 
handler includes persistent software components which extend the capabilities of a software 
10 application or an operating system to present media objects of a particular media type to an 
individual and autonomous software components, such as Java applets, which may only 
, B temporarily extend the capabilities of the host to present media. In addition, the term media 
O handler includes software applications that generate media experiences in real-time, such as 
fy video games, and resident software components, such as the PointCast™ agent, which present 
jjj|> media to end users. In addition, the term media handler includes those portions of an Internet 

% i browser, also called "viewers" and "plug-ins", that are capable and responsible for decoding 

fij 

1" specific media types, such as JPEG images, and using the resources of their host to present the 
media to the end user. The media handlers may be mobile, moving from one host computer to 
111 another, or stable, anchored to one host. 

gp Typically, an Internet browser software product has a collection of internal media 

handlers, such as JPEG and GIF decoders. When a browser encounters a media object of a given 
media type, the browser transfers the media object to the appropriate media handler for decoding. 
In addition, most browsers incorporate "plug-in" architecture, which allows third parties to 
develop new media types and distribute corresponding media handlers as "plug-ins." For 

2 5 example, Netscape Navigator™ currently has over two hundred compatible plug-ins available. 

It is noted that for continuous media, such as audio and video, identification tags 
are preferably placed at periodic intervals throughout the duration of the media or on an 
associated data channel. Since the identification tags might be utilized by hostile software to 
automatically remove advertising, the identification tags are preferably placed in both the 

30 advertising, as well as the associated advertising-supported media objects to discourage such 
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automatic removal. In one preferred embodiment, discussed below in conjunction with FIG. 12, 
real identification tags are placed in media that is to be measured (and a fraction that is not), and 
dummy identification tags are placed in other content. As discussed further below, in order to 
permit the research data collection agents 1500 to be tasked by the media research controller 300 
5 to report only registered media, the research data collection agent 1500 can preferably distinguish 
real identification tags from dummy identification tags, and, if desired, return only real 
identification tags to the media research controller 300. 

MEDIA RESEARCH CONTROLLER 
FIG. 3 is a block diagram showing the architecture of an illustrative media 
1 0 research controller 300. The media research controller 300 preferably includes certain standard 
hardware components, such as a central processing unit (CPU) 310, a random access memory 
, K (RAM) 320, a read only memory (ROM) 330, a clock 340, a communications port 350, and a 
G data storage device 360. The CPU 310 is preferably linked to each of the other listed elements, 
ffj either by means of a shared data bus, or dedicated connections, as shown in FIG. 3. 
f| The CPU 310 may be embodied as a single commercially available processor, 

such as Intel's Pentium 100 MHz P54C microprocessor, Motorola's 120 MHz PowerPC 604 
microprocessor or Sun Microsystem's 166 MHz UltraSPARC-I microprocessor. Alternatively, 
the CPU 310 may be embodied as a number of such processors operating in parallel, on one or 
more distributed processing nodes. The data storage device 360 and/or ROM 330 are operable to 
store one or more instructions, as discussed below in conjunction with FIGS. 12 through 14, 
which the CPU 310 is operable to retrieve, interpret and execute. The CPU 310 preferably 
includes a control unit, an arithmetic logic unit (ALU), and a CPU local memory storage device, 
such as, for example, an instruction cache or a plurality of registers, in a known manner. The 
control unit is operable to retrieve instructions from the data storage device 360 or ROM 330. 

2 5 The ALU is operable to perform a plurality of operations needed to carry out instructions. The 

CPU local memory storage device is operable to provide high-speed storage used for storing 
temporary results and control information. 

As discussed further below in conjunction with FIGS. 5 through 8, the data 
storage device 360 includes a central database 370 for storing a panel member database table 

3 0 500, a registered media database table 600, an encryption keys database table 700 and a log entry 
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database table 800. The panel member database table 500 preferably stores personal and 
demographic information for each member of the panel. The registered media database table 600 
preferably stores information about each media object that is registered with the media research 
controller 300, including the assigned tag identification number. The encryption keys database 
table 700 preferably stores the public key/private key pairs that are utilized in the illustrative 
embodiment to implement secure communications. The log entry database table 800 preferably 
stores information that has been extracted from or obtained about media objects presented to 
panel members and provided to the media research controller 300 by the remote media research 
agents 1500. 

In addition, as discussed further below in conjunction with FIGS. 12 through 14, 
the data storage device 360 preferably includes a registration and tag encoding process 1200, a 
data importing process 1300 and a reporting process 1400. Generally, the registration and tag 
encoding process 1200 registers a particular media object, and then places an assigned tag 
identification number in the registered media object, or otherwise associates the tag identification 
number with the media object. The data importing process 1300 receives log dispatches from the 
remote media research agents 1500, verifies the authenticity and integrity of the received 
messages and places confirmed entries in the log entry table 800. The reporting process 1400 
preferably generates reports in real-time or historically, to suit the needs of a particular research 
customer 150. 

The communications port 350 connects the media research controller 300 to the 
external networks 130, thereby linking the media research controller to each remote media 
research agent, as shown in FIGS. 1 and 2. The communications port 350 preferably includes 
multiple communication channels for simultaneously connecting the media research controller 
300 to multiple research data collection agents. 

PANEL MEMBER-COMPUTING DEVICE 

FIG. 4 is a block diagram showing the architecture of an illustrative panel 
member-computing device 400. The panel member-computing device 400 may be embodied as 
any device which presents media to individuals, including, for example, an Internet-enabled 
device, such as a network computer, a set-top box, a television, a telephone, pager, or personal 
digital assistant. The panel member-computing device 400 typically includes certain standard 
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hardware components, such as a central processing unit (CPU) 410, a random access memory 
(RAM) 420, a read only memory (ROM) 430, a clock 440, a communications port 450, and a 
data storage device 460. Each of these components 410, 420, 430, 440, 450 and 460 may be 
identical to the corresponding components described above in conjunction with FIG. 2. In 
5 addition, the panel member-computing device 400 may include one or more output devices 470, 
such as a video display card and video monitor, a sound card and speaker, and one or more 
multimedia players 480, such as a CD-ROM or DVD device. 

As discussed further below in conjunction with FIGS. 9 through 11, the data 
storage device 460 includes a local panel member database table 900, a local encryption keys 
10 database table 1000 and media queues 490. The local panel member database table 900 
preferably stores an indication of whether each panel member associated with a given research 
2 data collection agent is currently within the scope of the agent. The local encryption keys 
0 database table 1000 preferably stores each of the public key values which are required by the 

ill :j 

jjl research data collection agent 1500 to determine if a given extracted tag is a real tag or a dummy 
tag and for secure communications with the media research controller 300. The media queues 
FU 490 preferably store information that has been extracted from or obtained about media objects 
I* presented to panel members and provided to the research data collection agent by the media 
I;! handlers. As discussed further below in conjunction with FIGS. 15a through 15f, the media 
H queues 490 preferably consist of an unfiltered media queue 1150, a filtered media queue 1170, 
W each containing log entry objects 1100, and a dispatch queue 1190, containing dispatch objects 
1130. 

In addition, as discussed further below in conjunction with FIGS. 15 through 17, 
the data storage device 460 preferably includes a research data collection (RDC) agent 1500, a 
RDC/cooperative media handler host 1600 and one or more cooperative media handlers 1700. 

2 5 The research data collection (RDC) agent 1500 preferably includes (i) a logging process 1510 to 
create log entry objects from transmissions received from the cooperative media handlers and 
places the log entry objects into an unfiltered media queue, (ii) a filtering process 1530 to 
confirm the integrity of the messages and filter out unnecessary log entry objects from the 
unfiltered media queue, using a decoding subroutine 1550 to create a filtered media queue, (iii) a 

30 create dispatch process 1580 to create dispatch objects using objects from the filtered media 



21 



PATENT 



Docket No. 1600-2 US2 



queue and that places created dispatch objects into a dispatch queue, and (iv) a dispatch process 
1595 to transmit dispatch objects from the dispatch queue to the media research controller, when 
resources are available. The RDC/cooperative media handler host 1600 preferably provides 
resources and services 1675 to hosted processes and executes a tracking process 1605 to notify 
the research data collection agents 1500 when a panel member enters or exits the agent's scope. 
The cooperative media handlers 1700 preferably receive and decode media objects and extract 
information from, or otherwise obtain information about, presented media objects, including 
identification tags, if present, for collection by research data collection agents 1500. 

It is noted that the research data collection agent process 1500 and related 
database tables 900, 1000, 1100 have been shown as part of the panel member-computing device 
400 for illustrative purposes only, and could be resident on a device physically remote from the 
panel member 205 in alternate embodiments, as previously indicated, such as part of the panel 
member's net-centric environment which is activated as a side effect of the person obtaining 
access to network resources, or resident or hosted within a wristwatch that the panel member 
wears. 

The communications port 450 connects the panel member-computing device 400 
to the external networks 130, thereby linking the computing device 400 to the media research 
controller 300 and content providers 110, 120, as shown in FIGS. 1 and 2. The communications 
port 450 preferably includes multiple communication channels for simultaneous connections. 

DATABASE TABLES 

FIG. 5 illustrates an exemplary panel member database table 500 that preferably 
stores personal and demographic information for each member of the panel. The panel member 
database table 500 maintains a plurality of records, such as records 505-520, each associated with 
a different panel member. For each panel member identified by a panel member identifier in 
field 540, the panel member database table 500 includes the panel member's name, sex, age, city 
and state in fields 545 through 565, respectively. In addition, the panel member database table 
500 includes an indication of the member's education level and income in fields 570 and 575, the 
associated research data collection agent in field 580, and the member's email address in field 
585. The panel member identifier stored in field 540 may be utilized, for example, to index the 
log entry table 800, discussed below in conjunction with FIG. 8. 
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FIG. 6 illustrates an exemplary registered media database table 600 which 
preferably stores information about each media object which is registered with the media 
research controller 300, including the assigned tag identification number. The registered media 
database table 600 maintains a plurality of records, such as records 605-620, each associated with 
5 a different registered media object. For each registered media object identified by a registered 
media identification number (Mi) in field 640, the registered media database table 600 includes 
an indication of the entity which registered the object, the associated agency and the media type 
in fields 645 through 655, respectively. In addition, an encrypted and digitally signed version of 
registered media identification number preferably serves as the identification tag, C2, and is 
10 recorded in field 660. Finally, the anticipated starting and ending dates for which the media 

object will be distributed are recorded in fields 665 and 670. 
1^ FIG. 7 illustrates an exemplary encryption keys database table 700 that preferably 

J| stores the public key/private key pairs which are utilized by the media research controller 300 in 
PJ the illustrative embodiment to implement encrypted communications with the various entities 
J| and other security features. In a preferred embodiment, the media research controller 300 
jj| generates public key/private key pairs and securely distributes the various public keys to the 

m 

b research data collection agents of the panel members. In some embodiments, the agents may be 
jjy provided with initial public keys prior to distributing the agent. Thus, the encryption keys 
j *f database table 700 maintains a plurality of records, such as records 705-720, each associated with 
U a different public key/private key pair. For each pair, identified by a key pair identifier in field 
740, the encryption keys database table 700 includes an indication of the key pair owner in field 
745, namely, the entity which holds the private key, such as the research data collection agent 
(RDCA) 1500 or the media research controller (MRC) 300. In addition, the encryption keys 
database table 700 includes the corresponding public key and private key values in fields 750 and 

2 5 755, respectively. 

FIG. 8 illustrates an exemplary log entry database table 800 which preferably 
stores information which has been extracted from or obtained about media objects presented to 
panel members and provided to the media research controller 300 by the research data collection 
agents, in a manner described further below. The log entry database table 800 maintains a 

3 0 plurality of records, such as records 805-820, each associated with a different log entry. For each 
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log entry identified by an entry identifier in field 830, the log entry database table 800 includes an 
identifier of the research data collection agent 1500, cooperative media handler 1700, and media 
handler host 1600' associated with the log entry object in fields 835, 840 and 842, respectively. 
In this manner, the research media controller 300 can generate custom reports to analyze a 
panel's exposure to and interactions with the media handlers 1700 and media handler hosts 1600' 
themselves, in addition to the underlying media objects of interest. In addition, each logged entry 
includes an identifier of the particular media object and panel member associated with the entry 
in fields 845 and 850, as well as date and time stamps in fields 855 and 860. Each entry 
preferably also includes any metadata, contextual information, presentation information, and user 
interactions that were received in the log entry object, in fields 865, 870, 875 and 880, 
respectively. 

As shown in FIG. 8, the metadata in field 865 may include program identification 
information and the corresponding media type. The contextual information in field 870 may 
include the URL and other information indicating the source of the corresponding media object, 
or the context in which it was presented to the panel member. The presentation information in 
field 875 may include the language and format in which the media object was presented to the 
panel member, or other information indicating how the objected was presented to the panel 
member, when alternative presentations are possible. Finally, the user interactions in field 880 
may specify how the panel member interacted with the object, including whether the panel 
member zoomed in on portions of the media object, or rotated the object, as well as the size of 
the media object and any user inputs. 

FIG. 9 illustrates an exemplary local panel member database table 900 which 
preferably stores a semaphore indicating whether each panel member associated with a given 
research data collection agent is currently within the scope of the agent. The local panel member 
database table 900 maintains a plurality of records, such as records 905-915, each associated with 
a different panel member. For each panel member identified by a panel member identifier in 
field 940, the local panel member database table 900 contains a semaphore in field 945 indicating 
whether or not the panel member is in scope. 

FIG. 10 illustrates an exemplary local encryption keys database table 1000 which 
preferably stores each of the encryption key values which are required by the research data 
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collection agent 1500 to determine if a given extracted tag is a real tag or a dummy tag and for 
secure communications with the media research controller 300. The local encryption keys 
database table 1000 maintains a plurality of records, such as records 1005-1020, each associated 
with a different encryption key. For each public key identified by a local key identifier in field 
5 1040, the local encryption keys database table 1000 includes a key pair identifier in field 1045 
and the key value in field 1050. 

FIG. 1 1 illustrates an exemplary log entry object 1 100 from the media queues 490, 
which preferably stores information which has been extracted from or obtained about a given 
media object presented to one or more panel members and provided to the research data 
1 0 collection agent 1500 by the cooperative media handlers 1700. The log entry object 1 100 consists 
of a plurality of elements, including a date/time stamp 1102, panel member interactions 1104, 

H media handler identifier 1106, presentation information 1108, identification tag 1110, panel 

O 

Q member identifier 1 1 12, contextual information 1 1 14, metadata 1116 and the media handler host 
identifier 1118. Each of these information types has been described above in conjunction with 

S$ FIG. 8. In addition, the log entry object 1100 can include a copy of the media object itself, if 

fU desired. 

:\ PROCESSES 

Hi As discussed above, the media research controller 300 preferably executes a tag 

m 

M= registration and encoding process 1200, shown in FIG. 12a, to register a particular media object, 

2t and to place an assigned tag identification number in the registered media object, or on a 
simultaneous channel. As illustrated in FIG. 12a, the media research controller 300 begins the 
processes embodying the principles of the present invention during step 1210 upon receipt of a 
request to register a particular media object. 

Thereafter, the media research controller 300 assigns a media identification key, 

25 Mi, during step 1215 and creates a record of the media object in the registered media database 
table 600. In one embodiment, the identification tags are based on an extended version of the 
ISCI standard. In a preferred embodiment, the media research controller 300 utilizes a doubly 
encrypted identification tag to protect the security of the media identification key, which is 
preferably never made available outside the media research controller 300. Thus, the assigned 

3 0 media identification key, Mi, is preferably strongly encrypted with a private key, Ki, during step 
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1220 to produce a payload, C\. Thereafter, the payload, Q, is digitally signed with a private key, 
K 2 , to create a digital signature, DSi, during step 1225 and then the digital signature, DSi, is 
prepended to the payload, Q, during step 1230 to produce a new message, M 2) shown in FIG. 
12b. It is noted that the new message, M 2 , may optionally include bit padding to increase C\ to a 
5 predefined length. Finally, the new message, M 2 , is encrypted with a private key, K 3 , during step 
1235 to produce a real tag, C 2 . For a more detailed discussion of suitable encryption and security 
techniques, see B. Schneier, Applied Cryptography (2d ed. 1997), incorporated by reference 
herein. 

Thereafter, the real tag, C 2 , is placed in the media object during step 1240, or on a 
10 simultaneous channel, and also placed in field 660 of the registered media database 600. The 
identification tag can be bound to the content in a variety of ways. Preferably, the advertisement 
or other media is represented as an object, and has been designed to implement the cooperative 
q approach associated with the present invention by reserving a field or sub-object for storing the 

SIS 

l|j identification tag. It is important that the identification tag does not interfere in any way with the 
fll normal use of the media by any media handler or media handler host that has not implemented 
flj the cooperative API described herein. Binding can be generally achieved since the most 
: . prevalent media file formats and datatypes support extension through user-defined chunks or 
ill objects. As a last resort, the identification tag might be stored within an embedded comment 
iy : field, into its URL, or even through steganography (hidden codes). 

ft Finally, the media object is released for distribution during step 1245, before 

program control ends during step 1250. 

As discussed above, dummy identification tags may be utilized in some content to 
prevent hostile software from automatically removing advertising media objects. To promote the 
use of dummy identification tags, they are preferably constructed simply as a random sequence of 

25 n bits, as shown in FIG. 12c. In an alternate embodiment, the functionality for generating both 
real and dummy identification tags and encoding the identification tags into the media objects 
can be performed by third parties, such as advertising agencies or commercial producers, 
including edit houses or production facilities. 

As discussed above, the media research controller 300 preferably executes a data 

3 0 importing process 1300, shown in FIG. 13, to receive log dispatches from the remote media 
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research agents 1500, verify the authenticity and integrity of the received messages and place 
confirmed entries in the log entry table 800. As illustrated in FIG. 13, the media research 
controller 300 executes the data importing process upon receipt of a log dispatch from a research 
data collection agent during step 1310. 

Thereafter, the data importing process verifies the authenticity and integrity of the 
dispatch during step 1320 and then decrypts the dispatch during step 1330. Finally, the data 
importing process places all confirmed entries from the dispatch in the log entry database table 
800 during step 1340 before program control terminates during step 1350. 

As previously indicated, the media research controller 300 preferably executes a 
real-time reporting process 1400, or a non-real-time reporting process 1400', to generate reports 
suited to the needs of a particular research customer 150. As shown in FIG. 14, the data 
importing process 1300, discussed above, will create records of data received from the research 
data collection agents 1500 in a central database 370, from which the reports will be generated. 
The reports will be formatted to present available data, in a conventional manner, according to 
the needs one or more research customers 150. The customers 150 can access the reports over 
the external networks 130, or the reports can be printed off-line and provided to the customer 
150, as would be apparent to a person of ordinary skill. 

RESEARCH DATA COLLECTION AGENT 

As shown in FIG. 15a, the research data collection agent 1500 preferably includes 
a logging process 1510, a filter process 1530, a create-dispatch object process 1580 and a 
dispatch process 1595, discussed further below in conjunction with FIGS. 15b, 15c, 15e and 15f, 
respectively. The logging process 1510 is executed by the research data collection agent 1500 to 
receive messages from cooperative media handlers 1700 and create entry objects in an unfiltered 
media queue 1150. The filter process 1530 is executed to filter out dummy tags and other 
unnecessary information from the unfiltered media queue 1150 to create a filtered media queue 
1170. Finally, the research data collection agent 1500 executes the create-dispatch object process 
1580 to place the entries from the filtered media queue 1170 into a dispatch queue 1190 for 
transmission to the media research controller 300, by the dispatch process 1195 when resources 
are available. It is noted that the research data collection agent 1500 may also receive 
information regarding specific transactions made by an associated panel member, from a 
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cooperative financial or end-user transaction process, as would be apparent to a person of 
ordinary skill, based on the disclosure herein. 

As previously indicated, the research data collection agent 1500 executes a 
logging process 1510, shown in FIG. 15b, to receive messages from the cooperative media 
handler 1700 and to create entries in the unfiltered media queue 1150. Thus, the logging process 
1510 is entered during step 1515 upon receipt by the research data collection agent 1500 of a 
message from a cooperative media handler 1700 by means of the host 1600 of the media handler 
1700, containing an extracted identification tag, C 2 \ if available, and other information. 
Thereafter, the logging process 1510 creates a log entry object 1100 with the identification tag 
and other received information in the unfiltered media queue 1150 during step 1520, which is 
added to the tail of the unfiltered media queue 1150 during step 1525 before program control 
terminates during step 1528. 

As discussed above, the research data collection agent 1500 periodically executes 
a filter process 1530, shown in FIG. 15c, to filter out dummy tags and other unnecessary 
information from the unfiltered media queue 1150 to create a filtered media queue 1170. As 
shown in FIG. 15c, the filter process 1530 performs a test during step 1532 to determine if the 
unfiltered media queue 1150 is empty. If it is determined during step 1532 that the unfiltered 
media queue 1150 is empty, then program control ends during step 1548. If, however, it is 
determined during step 1532 that the unfiltered media queue 1150 is not empty, then a log entry 
object 1 100 is retrieved from the head of the unfiltered media queue 1 150 during step 1534. 

Thereafter, a test is performed during step 1535 to determine whether the media 
research controller 300 has instructed the research data collection agent 1500 to return 
information only about registered media. If it is determined during step 1535 that the media 
research controller 300 has not instructed the research data collection agent 1500 to return 
information only about registered media, then program control proceeds to step 1540. If, 
however, it is determined during step 1535 that the media research controller 300 has instructed 
the research data collection agent 1500 to return information only about registered media, then a 
decoding subroutine 1550, discussed below in conjunction with FIG. 15d, is executed during step 
1536 to determine if the received tag, C 2 ' , stored in the current entry object 1 100 of the unfiltered 
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media queue 1150, contains a real identification tag, which is generally associated with registered 
media. 

After the subroutine 1550 executes, a test is performed during step 1538 to 
determine if the received tag, C2', stored in the current log entry object 1100, contains a real 
identification tag. If it is determined during step 1538 that the received tag, C2', stored in the 
current log entry object 1100, does not contain a real identification tag, then program control 
returns directly to step 1532 to process additional log entry objects 1 100, if any. If, however, it is 
determined during step 1538 that the received tag, C 2 \ stored in the log entry object 1100, does 
contain a real identification tag, then the current log entry object 1100 is placed in the filtered 
media queue 1170 during step 1540. Thereafter, program control returns to step 1532 to process 
additional log entry objects 1 100, if any, in the manner described above. 

In this manner, for efficiency purposes, if the research data collection agent 1500 
is tasked by the media research controller to report only registered media, then the research data 
collection agent 1500 preferably only returns log entry objects 1100 containing real identification 
tags to the media research controller 300. The research data collection agent 1500 preferably 
does not know, however, whether or not the media object containing a real identification tag is an 
advertisement. As previously indicated, real identification tags are preferably placed in a small 
percentage of the advertising-subsidized media objects as a countermeasure against hostile 
advertising filtering software. Thus, if the research data collection agent's local encryption keys 
K2 and K3 are compromised and obtained by advertising filtering software, the advertising 
filtering software erroneously filters out some non-advertising media objects. It is noted that 
advertising filtering software is further discouraged by the preferred selection of challenging 
encryption techniques, since the research data collection agent 1500 can evaluate the authenticity 
of a received tag hours after the media is presented, while the advertising filtering software 
typically would need to decrypt the identification tags in real-time. 

As previously indicated, the filter process 1530 executes a decoding subroutine 
1550, shown in FIG. 15d, during step 1536 to determine if the received tag, C 2 \ stored in the 
current record of the unfiltered media log 1100, contains a real identification tag. The decoding 
subroutine 1550 is entered during step 1554, where the received tag, C2', is decrypted to create a 
message, M 2 \ using the value of the public key, K 3 , stored in the local encryption keys database 
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table 1000. The decoding subroutine 1550 then creates a digital signature, DSi\ from the first m 
bits of the message, M 2 \ and creates C\ from the remainder during step 1558. 

Thereafter, a test is performed during step 1562 to determine if the digital 
signature, DSi', is a valid digital signature of CY. If it is determined during step 1562 that the 
5 digital signature, DSi', is a valid digital signature of CY then the decoding subroutine returns an 
indication during step 1564 to the filter process 1530 that the received tag is a real identification 
tag. If, however, it is determined during step 1562 that the digital signature, DSi', is not a valid 
digital signature of CY then the decoding subroutine returns an indication during step 1568 to the 
filter process 1530 that the received tag is an invalid or dummy identification tag. In order to 
10 implement the security algorithms discussed in FIG. 15d, the research data collection agent 1500 
preferably has access to the necessary cryptographic services through its host, and has received 
the required public keys, K 2 and K 3 , by means of a secure distribution method from the media 

fas 

O research controller 300. 

fls 

ftf As discussed above, the research data collection agent 1500 executes a create- 

f| dispatch object process 1580, shown in FIG. 15e, to place entries from the filtered media queue 

m 

2$ 1170 into a dispatch queue 1190 for transmission by the dispatch process 1190, shown in FIG. 
s 15f, to the media research controller 300, when resources are available. Initially, the create- 
Ejj dispatch object process 1580 performs a test during step 1581 to determine if the filtered media 
pi queue 1170 is empty. If it is determined during step 1581 that the filtered media queue 1170 is 
|gp empty, then program control ends during step 1582. 

If, however, it is determined during step 1581 that the filtered media queue 1170 
is not empty, then a further test is performed during step 1583 to determine if a dispatch object 
1130 already exists. If it is determined during step 1583 that a dispatch object 1130 already 
exists, then program control proceeds directly to step 1585. If, however, it is determined during 

2 5 step 1583 that a dispatch object 1130 does not already exist, then a new dispatch object 1130 is 

created during step 1584. 

Thereafter, a log entry object 1100 is retrieved from the head of the filtered media 
queue 1150 during step 1585, and placed in the dispatch object 1130. Thereafter, a test is 
performed during step 1587 to determine if the filtered media queue 1170 is empty or if the 

3 0 dispatch object 1130 is full. If it is determined during step 1585 that the filtered media queue 
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1170 is not empty and that the dispatch object 1130 is not full, then program control returns to 
step 1581 to continue processing additional log entry objects 1100 from the filtered media queue 
1170, if any. 

If, however, it is determined during step 1585 that the filtered media queue 1170 
is empty or that the dispatch object 1130 is full, then the dispatch object 1130 will be compressed 
and digitally signed during step 1588. Thereafter, the compressed dispatch object 1130 will 
preferably be encrypted during step 1589 using the public key, K4, of the media research 
controller 300. Finally, the compressed and encrypted dispatch object 1130 will be added to the 
tail of the dispatch queue 1190 during step 1590 and program control returns to step 1581 and 
continues in the manner discussed above. 

As previously indicated, a dispatch process 1595, shown in FIG. 15f, transmits 
dispatch objects 1130 from the dispatch queue 1190 to the media research controller 300, when 
resources are available. Thus, a test is initially performed during step 1596 to determine if the 
dispatch queue 1190 is empty. If it is determined during step 1596 that the dispatch queue 1190 
is empty, then program control terminates during step 1597. If, however, it is determined during 
step 1596 that the dispatch queue 1 190 is not empty, then a dispatch object 1 130 is removed from 
the head of the dispatch queue 1190 during step 1598 and sent to the media research controller 
300 during step 1599. Thereafter, program control returns to step 1596 to process additional 
dispatch objects 1130, if any, in the manner described above. 

HOST PROCESSES) 

As previously indicated, the RDC/cooperative media handler host 1600 preferably 
executes a tracking process 1605, shown in FIG. 16, to notify the hosted research data collection 
agents 1500 when a panel member enters or exits the agent's scope. In addition, as discussed 
above, the research data collection agents 1500 and the cooperative media handlers 1700 
preferably rely on services and resources provided by the host processes for processing, security, 
storage and communication. In the illustrative embodiment discussed herein, an Internet browser 
software product, such as Netscape Navigator™ or Microsoft Internet Explorer™, resident on the 
panel member-computing device 400, serves as the host for both the research data collection 
agent 1500 and the cooperative media handler 1700 and provides them with computational 
resources. 
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Thus, as shown in FIG. 16, an illustrative tracking process 1605 continuously 
reports when a panel member enters or exits its scope during step 1610. Once the tracking 
process 1605 determines that a panel member has entered the scope of a research data collection 
agent, a message is sent to the hosted research data collection agent 1500 during step 1620 
monitoring the panel member indicating that a panel member has entered the agent's scope. 

Thereafter, the tracking process 1605 continuously monitors the panel member 
during step 1630 until the panel member leaves the scope of the hosted research data collection 
agent 1500. If it is determined during step 1630 that the panelist has left the scope of the hosted 
research data collection agent 1500, then a message is sent to the hosted research data collection 
agent 1500 during step 1640 indicating that a panel member has left the agent's scope. A test is 
then performed during step 1645 to determine if tracking should be suspended. If it is 
determined during step 1645 that tracking should not be suspended, then program control returns 
to step 1610 to continue tracking, in the manner described above. If, however, it is determined 
during step 1645 that tracking should be suspended, then program control terminates during step 
1650. 

COOPERATIVE MEDIA HANDLERS 

As discussed above, the cooperative media handler process 1700, shown in FIG. 
17, preferably presents media objects and extracts information from, or otherwise obtains 
information about, presented media objects, including identification tags, if present, for 
collection by research data collection agents 1500, whether or not a research data collection agent 
1500 is actually present. In an alternate implementation, the cooperative media handler 1700 can 
transmit all information obtained about a media object directly to the media research controller 
300 and perform all other tasks associated with the research data collection agents 1500. In other 
words, the cooperative media handler 1700 can directly serve as the software agent of the media 
research controller 300, without the need of an intermediary research data collection agent 1500, 
as would be apparent to a person of ordinary skill. 

In one embodiment, the media handler 1700 is provided with one or more 
remotely configurable settings which may be dynamically specified by the research data 
collection agent 1500 or the media research controller 300 to help filter out information which is 
not of interest to the research data collection agent 1500. Alternatively, the media handler 1700 
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can query its host to identify the information of interest to the research data collection agent 
1500. If, however, the host does not want to inform the media handler 1700 that no research 
data collection agent 1500 is present, the host might have the media handler 1700 pass 
everything. Thus, the media handler 1700 preferably reports information according to 
5 instructions received from the research data collection agent 1500 via its host. 

As shown in FIG. 17, a cooperative media handler 1700 is initiated by its host 
1600', such as the illustrative Internet browser, upon receipt of a media object having a media 
type that is compatible with the media handler. Thereafter, the media handler process 1700 
decodes and parses the media object during step 1705, in a known manner. A test is then 
10 performed during step 1710 to determine if there is media object information associated with the 
media object, such as an identification tag, metadata or contextual information. As previously 
** indicated, metadata may include program identification information and the corresponding media 
3 type and contextual information may include the URL and other information indicating the 
source of the corresponding media object, or the context in which it was presented to the end 
user. If it is determined during step 1710 that there is media object information associated with 
the media object, then a test is performed during step 1785 to determine if a message has already 
been created for the research data collection agent 1500. If it is determined during step 1785 that 
a message has already been created for the research data collection agent 1500, then program 
control proceeds directly to step 1795. If, however, it is determined during step 1785 that a 
g : 0 message has not yet been created for the research data collection agent 1500, then a message is 
created during step 1790. 

Thereafter, the media object information identified during step 1710 is added to 
the agent message during step 1795. Thereafter, a further test is performed during step 1720 to 
determine if there is media to present to a user. If it is determined during step 1720 that there is 
25 no media to present to a user, then program control proceeds directly to step 1740. If, however, it 
is determined during step 1720 that there is media to present to a user, then a conventional 
presentation process is performed during step 1725 to present the media. A test is then 
performed during step 1730 to determine if there is presentation information to report to the 
research data collection agent 1500. As previously indicated, presentation information may 
3 0 include the language and format in which the media object was presented to the panel member, 
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or other information indicating how the object was presented to the panel member, when 
alternative presentations are possible. If it is determined during step 1730 that there is no 
presentation information to report to the research data collection agent 1500, then program 
control proceeds directly to step 1760. If, however, it is determined during step 1730 that there is 
presentation information to report to the research data collection agent 1500, then the 
presentation information identified during step 1730 is added to the message during step 1735. 

A test is performed during step 1740 to determine if user interactions are received 
during the presentation of the media object. If it is determined during step 1740 that user 
interactions are received during the presentation of the media object, then program control 
proceeds directly to step 1760. If, however, it is determined during step 1740 that user 
interactions are received during the presentation of the media object, then the user interactions 
are received and processed during step 1745. A test is then performed during step 1750 to 
determine if there is user interaction information to report to the research data collection agent 
1500. As previously indicated, user interaction information may specify how the panel member 
interacted with the object, including whether the panel member zoomed in on portions of the 
media object, or rotated the object, as well as the size of the media object and any user inputs. 

If it is determined during step 1750 that there is no user interaction information to 
report to the research data collection agent 1500, then program control proceeds directly to step 
1760. If, however, it is determined during step 1750 that there is user interaction information to 
report to the research data collection agent 1500, then the user interaction information identified 
during step 1750 is added to the message during step 1755. 

A test is performed during step 1760 to determine if the presentation of the media 
object is finished. If it is determined during step 1760 that the presentation of the media object is 
not finished, then program control will return to step 1705 to continue processing in the manner 
described above. If, however, it is determined during step 1760 that the presentation of the media 
object is finished, then a test is performed during step 1765 to determine if there is a message to 
be sent to the research data collection agent 1500. If it is determined during step 1765 that there 
is a message to be sent to the research data collection agent 1500, then the message is completed 
during step 1770, for example, by adding media handler identifying information, and a final time 
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and date stamp. Thereafter, the completed message is sent to the research data collection agent 
1500. Program control will then terminate during step 1780. 

It is to be understood that the embodiments and variations shown and described 
herein are merely illustrative of the principles of this invention and that various modifications 
may be implemented by those skilled in the art without departing from the scope and spirit of the 
invention. 
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